A problem with managing a large amount of data is difficulty in providing sufficient storage for the data. For example, storing all collected network performance data in large data collection systems requires a vast amount of storage space. At the same time, since some of the collected data is never used in further processing, in practice, storing all collected network performance data may be unnecessary.
Another problem with managing a large amount of data is complexity of organizing the data for a future use. For example, indexing and storing the large amount of data may be a complex and time consuming process.
Still another problem with managing a large amount of data involves processing search queries issued to large databases. Due to the fact that large databases are often implemented as distributed databases with complex indexing schemes, generating execution plans for search queries issued to such databases and executing the execution plans may be a time consuming and inefficient process.
Yet another problem with a large amount of data is the impact of collecting, storing and processing the data upon data traffic in a network. Collecting, storing and processing the large amount of data require transmitting a great deal of data between nodes in the network, and thus may significantly impact communications bandwidth in the network and the speed in which the nodes communicate with each other.